Deep learning models, particularly Transformers, have achieved impressive results in various domains, including time series forecasting. While existing time series literature primarily focuses on model architecture modifications and data augmentation techniques, this paper explores the training schema of deep learning models for time series; how models are trained regardless of their architecture. We perform extensive experiments to investigate the occurrence of deep double descent in several Transformer models trained on public time series data sets. We demonstrate epoch-wise deep double descent and that overfitting can be reverted using more epochs. Leveraging these findings, we achieve state-of-the-art results for long sequence time series forecasting in nearly 70% of the 72 benchmarks tested. This suggests that many models in the literature may possess untapped potential. Additionally, we introduce a taxonomy for classifying training schema modifications, covering data augmentation, model inputs, model targets, time series per model, and computational budget.

Deep learning is a subset of machine learning, which in turn is a subset of artificial intelligence, but the origins of these names arose from an interesting history. In addition, there are fascinating technical characteristics that can differentiate deep learning from other types of machine learning... essential working knowledge for anyone with ML, DL, or AI in their skillset. If you are looking to improve your skillset or steer business/research strategy in 2021, you may come across articles decrying a skills shortage in deep learning. A few years ago, you would have read the same about a shortage of professionals with machine learning skills, and just a few years before that, the emphasis would have been on a shortage of data scientists skilled in "big data." Likewise, we've heard Andrew Ng telling us for years that "AI is the new electricity," and the advent of AI in business and society is constantly suggested to have an impact similar to that of the industrial revolution.

This video explores the recent study on Deep Double Descent! This is an interesting phenomenon of increasing and decreasing test error with respect to scaling up model size, training data, and epochs. The lens of bias-variance tradeoff may lead you to interpret increasing test error as overfitting, however the double descent phenomenon shows remarkable cases of a second descent when continuing to increase model size / epoch count. Awareness of this phenomenon may help you interpret your training curves for hyperparameter optimization and architectures search!

We show that the double descent phenomenon occurs in CNNs, ResNets, and transformers: performance first improves, then gets worse, and then improves again with increasing model size, data size, or training time. This effect is often avoided through careful regularization. While this behavior appears to be fairly universal, we don't yet fully understand why it happens, and view further study of this phenomenon as an important research direction. The peak occurs predictably at a "critical regime," where the models are barely able to fit the training set. As we increase the number of parameters in a neural network, the test error initially decreases, increases, and, just as the model is able to fit the train set, undergoes a second descent.